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^Sj ■ Abstract 

5— ( , We consider a property of positive polynomials on a compact set with a small 

^ ' perturbation. When applied to a Polynomial Optimization Problem (POP), 

the property implies that the optimal value of the corresponding SemiDefi- 
^— N ' nite Programming (SDP) relaxation with sufficiently large relaxation order is 

fT^ , bounded from below by (/* — e) and from above by /* + e{n + 1), where /* is 

the optimal value of the POP. We propose new SDP relaxations for POP based 
on modifications of existing sums-of-squares representation theorems. An ad- 
vantage of our SDP relaxations is that in many cases they are of considerably 
smaller dimension than those originally proposed by Lasserre. We present some 

(~| ' applications and the results of our computational experiments. 

-)— » ' 
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1 Introduction 

. ■ 1.1 Lasserre's SDP relaxation for POP 

> 

^O ■ We consider the POP: 

O 
O 



minimize f{x) subject to fi{x) > {i — 1, . . . , m), (1) 



■"sj" . where f,fi,---,fm'- IR" —^ K are polynomials. The feasible region is denoted hy K — {x ^W^ : fj{x) > 

^^ ' {j = 1, . . . ,m)}. Then it is easy to see that the optimal value /* can be represented as 



r =sup{p : f{x)~p>0{yxEK)}. 



First, we briefly describe the framework of the SDP relaxation method for POP ([T]) proposed by 
Lasserre [H]. See also [53]. We denote the set of polynomials and sums of squares by R[a;] and E, 
C^ , respectively. M[a:]r is the set of polynomials whose degree is less than or equal to r. We let E^ = EnR[a;]2r- 

We define the quadratic module generated by /i , . . . , /m as 






Mifl, ...Jra)^<^ O-O + 2^ O-j/j : cro, ■ • • , CTm e E 

The truncated quadratic module whose degree is less than or equal to 2r is defined by 

Mrifl, . . . , /m) = < CTQ + X! '^■J'/j' ■ "^O ^ ^'■' '^J ^ ^^J (J = 1, ■ • ■ , to) } 

where tj — r — [deg /j/2] for j = 1, . . . , rn. 
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Replacing the condition that f{x) — p is nonnegative by a relaxed condition that the polynomial is 
contained in Mr{fi, . . . , /,„), we obtain the following SOS relaxation: 

Pr^snp{p : f{x) - pe Mrifi, ..., fm) } ■ (2) 

Lasserred^ showed that p^ — >■ /* as r — ;> oo if M(/i, . . . , /„) is Archimedean. See [101 HI] for a 
definition of Archimedean. An easy way to ensure that M(/i, . . . , f„i) is Archimedean is to make sure 
that M{fi, . . . , fm) contains a representation of a ball of finite (but possibly very large) radius. In 
particular, we point out that when M{fi, . . . , /„,,) is Archimedean, K is compact. 

The problem ([2|) can be encoded as an SDP problem. Note that we can express a sum of squares 
(7 G Er by using a positive semidefinite matrix X e §^ as (j{x) — Ur{x)'^ Xur{x), where s{r) — ("+'"') 
and Ur{x) is the monomial vector which contains all the monomials in n variables up to and including 
degree r with an appropriate order. By using this relation, the containment by Mr{fi, . . . , fm) constraints 
in dH, i.e., 

m 

f - P = (jf) + ^(jjfj, 

can be transformed to linear equations involving semidefinite matrix variables corresponding to ctq and 

Note that, in this paper, we neither assume that K is compact nor that M{fi, . . . , /,„) is Archmedean. 
Still, the framework of Lasserre's SDP relaxation described above can be applied to (H]), although the 
good theoretical convergence property may be lost. 

1.2 Problems in the SDP relaxation for POP 

Since POP is NP-hard, solving POP in practice is sometimes extremely difficult. The SDP relaxation 
method described above also has some difficulty. A major difficulty arises from the size of the SDP 
relaxation problem ^. In fact, ^ contains ("^ '^) variables and s{r) x s{r) matrix. When n and/or r 
get larger, solving ([2|) can become just impossible. 

To overcome this difficulty, several techniques, using sparsity of polynomials, are proposed. See, e.g., 
[131 [T71 [501 1121 HZ]- Based on the fact that most of the practical POPs are sparse in some sense, these 
techniques exploit special sparsity structure of POPs to reduce the number of variables and the size of 
the matrix variable in the SDP ([1]). 

Another problem with the SDP relaxation is that ^ is often ill-posed. In [01[ini[2I], strange behaviors 
of SDP solvers are reported. Among them is that an SDP solver returns an 'optimal' value of (HI) which 
is significantly different from the true optimal value without reporting any numerical errors. Even more 
strange is that the returned value by the SDP solver is nothing but the real optimal value of the POP 
([1]). We refer to this as a 'super-accurate' property of the SDP relaxation for POP. 

1.3 Contribution of this paper 

POP contains very hard problems as well as some easier ones. We would like an approach which will 
exploit the structure in the easier instances of POP. In the context of current paper the notion of "easiness" 
will be based on sums of squares certificate and sparsity. Based on the next theorem and its variants, we 
propose new SDP relaxations. Our SDP relaxations can be interpreted as relaxations of those originally 
proposed by Lasserre. As a result, the bounds generated by our approach cannot be superior to those 
generated by Lasserre's approach for the same order relaxations. However, our SDP relaxations are 
of significantly smaller dimension (compared to Lasserre's SDP relaxations) and as the computational 
experiments in Section 5 indicate, we obtain very significant speed-up factors and we are able to solve 
larger instances and higher-order SDP relaxations. Moreover, in most cases, the amount of loss in the 
quality of bounds is small, even for the same order SDP relaxations. 
We assume that there exists an optimal solution x* of ([T]). Let 

b = max (1, max{ |x*| : i = !,...,«}) 
B = [-6,6]". 



Obviously x* e B. We define: 

K = Br\K 

Rj = niax{ |/j(a;)| : x e B] (j = 1, , 

Define also, for a positive integer r, 



Mx) = 




R, ) 


e,(x) = 






e.,,,(a:) = 


-g(r- 





We start with the following theorem. 
Theorem 1 Suppose that for p e R, I{x) — p > for every x € K, i.e., p is a lower bound of f* . 
i. Then there exists f G N such that for all r > f, f — p + tpr is positive over B. 
ii. In addition, for every e > 0, there exists a positive integer f such that, for every r > f, 

f - p + eOr;b + '4>f e S. 

Theorem [T] will be proved in Section [5] as a corollary of Theorem [5] We remark that r depends 
on p and e, while f depends on p, but not e. The implication of this theorem is twofold. First, it 
elucidates the super-accurate property of the SDP relaxation for POPs. Notice that by construction, 
—i/jfix) (z Mf{fi, . . . , fm) where f = fmaxj(deg(/j)). Now assume that in ^, r > f. Then, for any 
lower bound p of /*, Theorem [1] means that f — p + e&rfi & Mr{fi, . . . , fm) for arbitrarily small e > 
and sufficiently large r. Such a small perturbation is inevitably introduced everywhere in the floating 
point arithmetic which is used by the interior-point methods for solving the SDP relaxations. Note that 
we chose an arbitrarily lower bound of /*, and in ([2]), the lower bound is being maximized. Therefore, 
we may obtain /* due to the implicit perturbation introduced by the floating point arithmetic. 

Second, we can use the result to construct new sparse SDP relaxations for POP H]). Our SDP relax- 
ation is weaker than Lasserre's, but the size of our SDP relaxation can become smaller than Lasserre's. 
As a result, for some large-scale and middle-scale POPs, our SDP relaxation can often obtain a lower 
bound, while Lasserre's cannot. 

A naive idea is that we use ([T]) as is. Note that —tprix) contains only monomials whose exponents are 
contained in 

where J^j is the support of the polynomial fj, i.e., the set of exponents of monomials with nonzero 
coefficients in fj, and JFj = J-j U {0}. To state the idea more precisely, we introduce some notation. For 
a finite set J-" C N" and a positive integer r, we denote rT = J- + ■ ■ ■ + J^ and 



-J-, + ---+J-, 



^(^)^{ £5fc(a^)' : supp(gfc)C^ 



fc=i 



where supp((7fc) is the support of g^. Note that S(J^) is the set of sums of squares of polynomials whose 
supports are contained in T. 



Now, fix an admissible error e > and f as in Theorem [TJ and consider: 

f " ~ \ 

/5(e, r, r) = sup < p : f - p + eOr.b - ^ IjCTj = o-q, ctq e ^r,crj G S(rJ'j) > (3) 

for some r > f. Due to Theorem [I] ([3]) has a feasible solution for all sufficiently large r. 
Theorem 2 For every e > 0, there exist f, r e N such that /* — e < p(e, f , r) < f* + e{n + 1). 

Proof : We apply Theorem [T] to POP (U) with p = f* — e. Then for any e > 0, there exist f, f G N such 
that for every r > f, f — (/* — e) + eOr.b + V'r G S. Choose a positive integer r > f which satisfies 

r > max{ rdeg(/)/2l , \{P + 1/2) deg(/i)l ,---,\ir + 1/2) deg(/„)l }. (4) 

Then there exists fJo € Sr such that / — (/* — e) + eOr.fc + ''Pf — ^o, because the degree of the polynomial 
in the left hand side is equal to 2r. We denote a-j := (1 — fj/Rj) ^ for all j. The triplet (/* — e, (Tq, CTj) 
is feasible in ^ because (1 — fj/Rj) ^ G ^i^^j)- Therefore, we have /* — e < p{e,r,r). 

We prove that p{e, f, r) < f* + e{n + 1). We choose r as in (H]) and consider the following POP: 

/:= mi {fix) + eer,bix):fiix)>0,...Jmix)>0}. (5) 

Applying Lasserre's SDP relaxation with relaxation order r to ([5]), we obtain the following SOS relaxation 
problem: 

f " 1 

p{e, r) := sup < p : f - p + eQrM = ^^o + ^ fj^ji^^o € Sr, (^j G S,.^. 5. , (6) 

where rj :— r — [dcg(/j)/2] for j = 1, . . . , to. Then we have /5(e, r) > p(e, f, r) because Yj{fJ-j) C S,.. for 
all j. Indeed, it follows from ^ and the definition of rj that rj > fdeg{fj), and thus I](fJv,) C E^ . 

Every optimal solution x* of POP ([!]) is feasible for ([5]) and its objective value is /* + Qr,b{x*). We 
have /* + Qr.bix*) > p{i,r) because ([6]) is the relaxation problem of ([5]). In addition, it follows from 
X* e B that n + 1 > Qr,b{x*), and thus p(e, f , r) < p{e, r) < f* + e(n + 1). D 

Lasserre |15| proved the convergence of his SDP relaxation under the assumption that the quadratic 
module M(/i, . . . , /„) associated with POP (H]) is Archimedean. In contrast, Theorem[5]does not require 
such an assumption and ensures that we can obtain a sufficiently close approximation to the optimal 
value /* of POP © by solving ©. 

We delete the perturbed part eOr,&(x) from the above sparse relaxation 1^ in our computations, 
because it may be implicitly introduced in the computation by using floating-point arithmetic. In the 
above sparse relaxation ([3]), we have to consider only those positive semidefinite matrices whose rows 
and columns correspond to fj^j for fj. In contrast, in Lasserre's SDP relaxation, we have to consider 
the whole set of monomials whose degree is less than or equal to rj for each polynomial fj . Only ao is 
large; it contains the set of all monomials whose degree is less than or equal to r. However, since the 
other polynomials do not contain most of the monomials of ao-, such monomials can safely be eliminated 
to reduce the size of ctq (as in |I3])- As a result, our sparse relaxation reduces the size of the matrix 
significantly if each \Tj\ is small enough. We note that in many of the practical cases, this in fact is true. 
We will call this new relaxation Adaptive SOS relaxation in the following. 

Finally, we consider the case where the feasible region K is empty. —1 S Af (/i, . . . , /,„) is a sufficient 
condition for K to be empty, but it is not necessary. Indeed, if — 1 G A/(/i, . . . , /,„), then there exist 
sums of square polynomials tJo, ■ • • i o'm such that —1 = ao{x) + X^^li '^jix)fj{x). If K is nonempty, then 
we obtain a contradiction by substituting x € K in this identity. However, — 1 G M(/i,...,/m) may 
not hold even if K is empty. For instance, let /i — x,f2 — y,f3 = —1 — xy. Then K is empty, but 
— 1 ^ A/(/i, /2, /a). We can prove this fact by using the discussion in [24j Example 6.3.1]. 

The following result is directly obtained as a corollary of Theorem [TJ We omit the proof. 



Corollary 3 Suppose K is empty. Then for every e > 0, there exists f G N such that for every r > f, 
-1 + ee, eM(/i,...,/„). 

The rest of this paper is organized as fohows. In Section [21 we give three extensions of Theorem [TJ 
During the course of the section, we prove Theorem [T] We mention some of the related work to Theorem 
[1] and present the proposed SDP relaxation in Section [3l Section |4] gives our proposed SDP relaxation 
based on Theorem [T] In Section [51 we present the results of some numerical experiments. 

2 Extensions 

In this section, we give three extensions of Theorem [l] 

2.1 Sums of squares of rational polynomials 

We can extend part i. of Theorem [T] with sums of squares of rational polynomials. We assume that for 
all j = 1, . . . , m, there exists gj S M.[x] such that |/j(a;)| < gj{x) and gj{x) ^ for all x E B. We define 

for all r G N. Then, we can prove the following corollary by using almost the same arguments as Theorem 

m 

Corollary 4 Suppose that for p G M, f{x) — p > for every x £ K, i.e., p is an lower bound of /*. 
Then there exists f G N such that for all r > f, f — p + 4'r ^s positive over B. 

It is difficult to apply Corollary [3] to the framework of SDP relaxations, because we deal with rational 
polynomials in ■0^. However, we may be able to reduce the degrees of sums of squares in -0^. by using 
Corollary[31 For instance, we consider fi{x) = \ — x^ and B = [—1, 1]. Choose gi{x) = 2(1 + x"^). Then 
gi dominates |/i| over B, i.e., |/i(a;)| < gi{x) for all x E B. We have 

and the degree of ip in Corollary [H is 4r, while the degree of tp in Theorem [T] is 8r. 

2.2 Extension to POP with correlative sparsity 

In [27], the authors introduced the notion of correlative sparsity for POP IJ), and proposed a sparse SDP 
relaxation that exploits the correlative sparsity. They then demonstrated that the sparse SDP relaxation 
outperforms Lasserre's SDP relaxation. The sparse SDP relaxation is implemented in [28] and its source 
code is freely available. 

We give some of the definition of the correlative sparsity for POP ([TJ. For this, we use an n x n 
symbolic symmetric matrix R, whose elements are either or * representing a nonzero value. We assign 
either or • as follows: 



Rk,i 



* if k^i, 

* if Qffc > 1 and ai > 1 for some a G J-", 

* if Xk and xi are involved in the polynomial fj for some _;/ = 1, . . . , m, 
o.w. 



POP H]) is said to be correlatively sparse if the matrix R is sparse. 

We give some of the details of the sparse SDP relaxation proposed in [57] for the sake of completeness. 
We construct an undirected graph G = {V, E) from R. Here V := {1, . . . , n} and E := {(/c, t) : Rk,e = *}■ 
After applying the chordal extension to G = {V, E), we generate all maximal cliques Ci, . . . , Cp of the 




extension G — (V, E) with E C E. See [HI ^7\ and references therein for the details of the construction 
of the chordal extension. For a finite set C C N, xc denotes the subvector which consists of Xi [i £ C). 
For all /i,...,/m in POP dH), Fj denotes the set of indices whose variables are involved in /j, i.e., 
Fj := {i £ {1, . . . , n} : tti > 1 for some a £ Tj}. For a finite set C C N, the sets 'Sr,c and Eoo,c denote 
the subsets of Er as follows: 

r 9 

^'%C := {y^,9k{xf ■■yk = l,...,q,gk£R[xc] 

r>0 

Note that if C = {1, . . . , n}, then we have Sr,c = S^ and Eoo.c = S. The sparse SDP relaxation problem 
with relaxation order r for POP ^ is obtained from the following SOS relaxation problem: 

sparse.^ f /-P = ELl'^OJi+Ej"Ll^j/j. 1 /^N 

where Dj is the union of some of the maximal cliques Ci , . . . , Cp such that Fj C Ch and r^ = r — 
rdeg(/,)/2l for j = l,...,m. 

It should be noted that other sparse SDP relaxations are proposed in [71 [T71 HD] and the asymptotic 
convergence is proved. In contrast, the convergence of the sparse SDP relaxation ([7]) is not shown in P7]. 

We give an extension of Theorem[T]to POP with correlative sparsity. If Ci , . . . , Cp C {1, . . . , n} satisfy 
the following property, we refer this property as the running intersection property (RIP): 

V/ie {!,..., p-l},3t£ {l,...,p} such that C,,+i n (Ci U • • • U C,,) C Cj. 

For Ci , . . . , Cp C {1, . . . , n}, we define sets Ji , . . . , Jp as follows: 

A := {je {!,..., m}:/,eR[xcJ}. 

Clearly, we have U^^iJh = {1, • • ■ , Tn}. In addition, we define 



A,h{^) ■= -^ /j(a 



jeJh 






e 



r.h,h 



:-) - '+,i:(fy 



ieCh 



for h = 1, . . . ,p. 

Using a proof similar to the one for the theorem on convergence of the sparse SDP relaxation given 
in [7j, we can establish the correlatively sparse case of Theorem[T] Indeed, we can obtain the theorem by 
using [7| Lemma 4] and Theorem [1] 

Theorem 5 Assume that nonempty sets Ci, . . . , Cp C {1, . . . , n} satisfy (RIP) and we can decompose f 
into f ~ fi + ■ ■ ■ + fp with fh £ M[xCh] {h — ^t ■ ■ ,p)- Under the assumptions of Theorem\^ there exists 
r € N such that for all r > f, f — p + X]h=i V'r,^ 'is positive over B = [—6, &]". In addition, for every 
e > 0, there exists f £ N .such that for all r > f, 

p p 

f - P + f^'^QrJi,b + ^->Pf,h £ Soo.Ci ^ ^ Soo, Cp- (8) 

h=l h=l 

Note that if p = 1, i.e., Ci = {1, . . . , n}, then we have ipr,i — ipr and Qr,i,b — &r,b, and thus Theorem[S] 
is reduced to Theorem [1] Therefore, we will concentrate our effort to prove Theorem [5] in the following. 
In addition, we remark that it would follow from [71 Theorem 5] that ([5]) holds without the polynomial 
e^^_-i^ Qr.h.b if we assume that all quadratic modules generated by fj {j £ Ch) for all h = 1, . . . ,p are 
Archimedean. 

To prove Theorem [51 we use Lemma 4 in J^ and Corollary 3.3 of [TO] , 



Lemma 6 ( modified version of /7, Lemma 4]) Assume that we decom,pose f into f ~ fi + ■ ■ ■ + fp with 
fh G M[a;Cft] and f > on K . Then, for any bounded set B C R", there exist f e N and gt G M[a:Ch] 
with gh > on B such that for every r > f, 

p p 

h=l h=l 

Remark 7 The original statement in f7| Lemma 4] "i-s slightly different from Lemma \^ Ln ^7^ Lemma 
4-], it is proved that there exists A G (0, 1], f G N and gt G R[xCft] with gh > on B such that 

h=lj£Jh h=l 

In AvvendixV^ we establish the correctness of Lemma\^by using J^ Lemma 4]- 

Lemma 8 (Corollary 3.3 of J19f) Let / G M[a;] be a polynomial nonnegative on [—1,1]". For arbitrary 
e > 0, there exists some f such that for every r > f, the polynomial (/ + eQr) is a SOS. 

Proof of Theorem\^ : We may choose [—6,6]" as B in Lemma [S] It follows from the assumption in 
Theorem [S] that we can decompose f — p into (/i — p) + /2 + • • • + /p- Since /i — p G M[a;ci], i* follows 
from Lemmainithat there exists f G N and gt G R[a;c,J with gh > on _B such that for every r > f, 

p p 

f-p^{fi-p) + f2 + --- + .fp = -J2 ^-^h + Y.9'^- 

h=l h=l 

Therefore, the polynomial f — p + ^^=1 V'r.ft, is positive on B for all r > f. 

For simplicity, we fix h and define Ch = {ci, . . . , Cfe}. Then, gh consists of the k variables Xci , • • ■ , Xc^. ■ 
Since gh > on B, it is also positive on B' := {{xci, • • ■ ,Xc^) : —b < Xcj < b {j — 1, . . . ,fc)}. We define 
9hiy) = 9h{by). Since gh is positive on B', gh G M[2/ci, • ■ • ,2/cJ is also positive on the set {(yci, • • ■ ,2/cfc) : 
— 1 < Vcj < 1 = 1, ■ • ■ , k)}. Applying Lemma |S]to gh, for all e > 0, there exists f/j G N such that for 
every r > fh, 

k 

QhiVc^,- ..,ycj + e^ylll^ crhiVc^,- ■ • ,2/cJ 
for some ah G Y^oo.c'h- Substituting Xci — byci, • • ■ ,Xc^ = byc^, we obtain 

gh + ^'drM,b G T,oo,Ch- 

We fix e > 0. Applying the above discussion to all /i = 1, . . . ,p, we obtain the numbers fi, . . . , fp. We 
denote the maximum over fi, . . . , fp by f . Then, we have 

p p 

/ - p + e ^ Qr.hfi + ^ 'ipf.h G Soo,Ci H 1- ^oo.Cp 

h=l h=l 

for every r > f. D 

2.3 Extension to POP with symmetric cones 

In this subsection, we extend Theorem [1] to POP over symmetric cones, i.e., 

f* := inf {fix) : G{x) G £+} , (9) 

where / G R[x], 5+ is a symmetric cone associated with an iV-dimensional Euclidean Jordan algebra £, 
and G is f -valued polynomial in x. The feasible region K of POP © is {x G R" : G{x) G £+}. Note that 



if £ is R™ and S+ is the nonnegative orthant M™, then ^ is identical to ([T]). In addition, S!j:, the cone 
of n X n symmetric positive semidefinite matrices, is a symmetric cone, the bihnear matrix inequalities 
can be formulated as ([9|). 

To construct il)r for (j9]), we introduce some notation and symbols. The Jordan product and inner 
product oi x,y G £ are denoted by, respectively, x o y and x • y. Let e be the identity element in the 
Jordan algebra £. For any x G £, we have eox = xoe — x. We can define eigenvalues for all elements 
in the Jordan algebra £, generalizing those for Hermitian matrices. See 4 for the details. We construct 
V'r for dn} as follows: 

M := sup {maximum absolute eigenvalue of G{x) : x € K^ , 

Mx) ■■= -G(^)'(e-^) '' (10) 

where we define x'^ :— x^^^ o x for fc e N and x <G £. 

Lemma 4 in [T3] shows that i/'r defined in (|10p has the same properties as ipr in Theorem [TJ 

Theorem 9 For a given p, suppose that f{x) — p > for every x £ K . Then, there exists f € N such 
that for all r > r, f — p + ipr is positive over B. Moreover, for any e > 0, there exists f e N such that 
for every r > f, 

f - p + e9.r,6 + f/'? G S. 

3 Related works 

3.1 Another perturbed sums of squares theorem 

In this subsection, we present another perturbed sums of squares theorem for POP ([1]) which is obtained 
by combining results in [T2j [16] . 

To use the result in 121, we introduce some notation and symbols. We assume that K C B := [—b, 6]". 
We choose 7 > 1 such that for all j — 0,1, . . . ,m, 



\fj{x)/j\ < lif ||x||oo< V2&, 
mx)h\ < \\x/b\\tii\\x\\^>V2b, 



where /o denotes the objective function / in POP dH), and d = max{deg(/), deg(/i), . . . , deg(/„)}. For 
r e N, we define 






X, 



From (a), (b) and (c) of Lemma 3.2 in [12], we obtain the following result: 

Proposition 10 Assume that the feasible region K of POP ([I]) is contained in B = [—b, 6]". In addition, 
we assume that for p G R, we have f ~ p > over K . Then there exists f G N such that for all r > f, 

if — p + ipr + 4'r,b) is positive over M". 

We remark that we do not need to impose the assumption on the compactness of K in Proposition 
[TUl Indeed, we can drop it by replacing K by K defined in Subsection 11.31 as in Theorem [T] 

Next, we describe a result from [16] which is useful in deriving another perturbed sums of squares 
theorem. 



Theorem 11 ((in) of Theorem 4.1 in fl6f) Let / G R[a;] be a nonnegative polynomial. Then for every 
e > 0, there exists r e N such that for all r >f, 

f + e9re E, 

where ^^(x) := J2ti J2l=oi^?/k^-)- 

By incorporating Proposition [TU] with Theorem 1111 we obtain yet another perturbation theorem. 

Theorem 12 We assume that for p e M, we have / — p > over K . Then we have 
i. there exists f g N such that for all r > f, {f — p + 4'r + 4'r.b) is positive over M"; 
ii. moreover, for every e > 0, there exists f e N such that for all r > f, 

if - P + i^f + 4>f,b + e^r) e s. 

We give an SDP relaxation analogous to ([3]), based on Theorem [T^ as follows: 

m n 

?7(e,r,r) :=sup<( r;: ~{ i^i (^ (^1) 

(To e S.r,crj e Y.{rFj),p.i e Y.{{d{f+ l)ei}) 

for some r > f, where e; is the ith standard unit vector in M". One of the differences between ^ and 
(|lip is that (|lip has n SOS variables /ii, . . . ,/i„. These variables correspond to nonnegative variables 
in the SDP formulation, but not positive semidefinite matrices, since these consist of a single monomial. 
On the other hand, it is difhcult to estimate f in the SDP relaxations ([3]) and (fTTj) . and thus we could 
not compare the size and the quality of the optimal value of ([3]) with (|lip so far. 

We obtain a result similar technique to Theorem [2j We omit the proof because we obtain the inequal- 
ities by applying a proof similar to that of Theorem [2] 

Theorem 13 For every e > 0, there exists r,f(EN such that f* — e< r/(e, f , r) < f* + ene^ . 

3.2 Other works 

We mention other research related to our work based on Theorem [Ij A common element in all of these 
approaches is to use perturbations e9r{x) or eQr{x) for finding an approximate solution of a given POP. 
In [HI [To], the authors added eOr(x) to the objective function of a given unconstrained POP and 
used algebraic techniques to find a solution. In TT , the following equality constraints were added in the 
perturbed unconstrained POP and Lasserre's SDP relaxation was applied to the new POP: 

§^ + 2rexr' = 0(z = l,...,n). 
oxi 

Lasserre in [18] proposed an SDP relaxation via 9^{x) defined in Theorem 1111 and a perturbation 
theorem for semi- algebraic set defined by equality constraints gk{x) — Q (k — 1, . . . ,m). The SDP 
relaxation can be applied to the following POP: 

inf {/o(x) : gk{x) = (fc = 1, . . . , m)} ; (12) 

To obtain the SDP relaxations, we add e9r(x) to the objective function in POP ([T2|) and replace the 
equality constraints in POP ([T2|) by gl{x) < 0. In the resulting SDP relaxations, 9r{x) is explicitly 
introduced and variables associated with constraints g1{x) < are not positive semidefinite matrices, but 
nonnegative variables. 



4 Adaptive SOS relaxation 

4.1 Constructing Adaptive SOS relaxation 

A SOS relaxation ([3]) for POP Q has been introduced in Section [1] However, this relaxation has some 
weak points. In particular, we do not know the value f in advance. Also, introducing small perturbation 
e intentionally may lead numerical difficulty in solving SDP. 

To overcome these difficulties, we ignore the perturbation part eQr.b{x) in ([3]) because the perturbation 
part may be implicitly introduced by floating point arithmetic. In addition, we choose a positive integer 
r and find f by increasing r. Furthermore, we replace Uj G T^lfTj) by Cj G Yi{fjJ-j) in (|31), where fj is 
defined for a given integer r as 

r 1 



.dcg(/,) 2_ 
to have deg(/jCTj) < 2r for all j = 1, . . . , m. Then, we obtain the following SOS problem: 

r m I 

p*(r) := sup _ < p: /-p-^/jCTj =cro ? • (13) 

pGR,o-oGSr,CTjeS(rj.Fj) I j—i I 

We call ([T^ Adaptive SOS relaxation for POP ((T|). Note that we try to use numerical errors in a positive 
way; even though Adaptive SOS relaxation has a different optimal value from that of POP, we may hope 
that the contaminated computation produces the correct optimal value of POP. 

In general, we have Ti{fjFj) C E,. because of rjdeg{fj) < tj. Recall that tj = r — [dcg(/j)/2] 
and is used in Lasserre's SDP relaxation ([2]). This implies that our SDP relaxation is no stronger than 
Lasserre's SDP relaxation, i.e., the optimal value p*{r) is lower than or equal to the optimal value p{r) 
of Lasserre's SDP relaxation for POP ([T]) for all r. We further remark that p*{r) may not converge the 
optimal value /* of POP H]). However, we can hope for the convergence of p*{r) to /* from Theorem [1] 
and some numerical results in HH |3I] • 

4.2 A property for a quadratic optimization problem 

We consider the quadratic optimization problems (QOPs): 

inf^ {f{x) := x^Pqx + c^x : /^(x) := x'^ PjX + cjx + 7j > (j = 1, . . . , m)} . (14) 

We assume that for QOP ((M]) , none of the symmetric matrices Pj is a zero matrix. This means that the 
degree deg(/j) = 2 for all j = 1, . . . , m. 

In this subsection, we prove that we have p*{r) = p*(r — 1) if r is even number. This implies that we 
do not need to compute p*{r) for even r. It follows from definition of fj that we have 



r- 1 



^^ if r is odd, 
— 1 if r is even. 



2 

We assume that r is even and give our SDP relaxation problems with relaxation order r and r — 1: 

p*{r) = supip:/-p-^/jO-j =o-o,peK,ffoeSr,o-j es((^-l) J-j) [, (15) 




p*(r-l) = sup<| p : /-p-^/jCTj = cro,pe M,(To e Sr-i,crj e S(^(^- - ij J'jj i . (16) 

We have p*{r) > p*{r — 1) for ([T5|) and ([TC)) . All feasible solutions (p, ctq; ""j") of p3|) satisfy the following 
identity: 

m 
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Since r is even, the degrees of X^j'li '^ji^)fj{^) ^^'^ foix) — p are less than or equal to 2r — 2 and 2 
respectively, and thus, the degree of ai^ is less than or equal to 2r — 2. Indeed, we can write (Jo{x) — 
Tlik^i idkix) + hk{x)) , where deg{gk) < r — 1 and hk is a homogenous polynomial with degree r. Then 
we obtain = J2k=i ^fc(^)' which implies hk = for all k = !,...,£. Therefore, all feasible solutions 
(p, ao, aj) in SDP relaxation problem p5|) are also feasible in SDP relaxation problem ([T6)) . and we have 
p*{r) = p*(r — 1) if r is even. 

5 Numerical Experiments 

In this section, we compare Adaptive SOS relaxation with Lasserre's SDP relaxation and the sparse 
SDP relaxation using correlative sparsity proposed in [271. To this end, we perform some numerical 
experiments. We observe from the results of our computational experiments that (i) although Adaptive 
SOS relaxation is often strictly weaker than Lasserre's, i.e., the value obtained by Adaptive SOS relaxation 
is less than Lasserre's, the difference is small in many cases, (ii) Adaptive SOS relaxation solves at least 
10 times faster than Lasserre's in middle to large scale problems. Therefore, we conclude that Adaptive 
SOS relaxation can be more effective than Lasserre's for large- and middle-scale POPs. We will also 
observe a similar relationship against the sparse relaxation in [27 ; Adaptive SOS relaxation is weaker 
but much faster than the sparse one. 

We use a computer with Intel (R) Xcon (R) 2.40 GHz cpus and 24GB memory, and Matlab R2010a. 
To construct Lasserre's [T5j, sparse [27^ and Adaptive SOS problems, we use SparsePOP 2.99 [21] • To 
solve the resulting SDP relaxation problems, we use SeDuMi 1.3 25 with the default parameters and 
SDPT3 4.0 [ini- The default tolerances for stopping criterion of SeDuMi and SDPT3 are l.Oe-9 and l.Oe- 
8, respectively. We check DIMACS errors for the quality of the approximate solution. If the six errors 
are sufficiently small, then the solution is regarded as an optimal solution. See [21j for the definitions. 

To determine whether the optimal value of an SDP relaxation problem is the exact optimal value of 
a given POP or not, wc use the following two criteria £q\^\ and efeag^ Let x be a candidate of an optimal 
solution of the POP obtained from the SDP relaxations. We apply a projection of the dual solution of 
the SDP relaxation problem onto M" for obtaining x in this section. See [27 for the details. We define: 

[the optimal value of the SDP relaxation — f(x)\ 
''obj - max{l,/(i)} ' 

If Cfgj^g — 0, then x is feasible for the POP. In addition, if Cq]-,] = 0, then x is an optimal solution of the 
POP and f{x) is the optimal value of the POP. 

We introduce the following value to indicate the closeness between the obtained values of Lasserre's, 
sparse and Adaptive SOS relaxations. 

. (obj. val. of Lasserre's or sparse SDP relax. ) p* 

'~ (obj. val. of Adaptive SOS relax.) ^ p*{r)' 

If the signs of both optimal values are the same and Ratio is sufficiently close to 1, then the optimal 
value of Adaptive SOS relaxation is close to the optimal value of Lasserre's. In general, this value is 
meaningless for measuring the closeness if those signs are different or either of values is zero. Fortunately, 
those values are not zero and those signs are the same in all numerical experiments in this section. 

To reduce the size of the resulting SDP relaxation problems, SparsePOP has functions based on the 
methods proposed in [T31 [32] . These methods are closely related to a facial reduction algorithm proposed 
by Borwein and Wolkowicz [TJ [5], and thus we can expect the numerical stability of the primal-dual 
interior-point methods for the SDP relaxations may be improved. In this section, except for Subsection 
15.11 we apply the method proposed in [35]. 

For POPs which have lower and upper bounds on variables, we can strengthen the SDP relaxations 
by adding valid inequalities based on these bound constraints. In this section, we add them as in [27] . 
See Subsection 5.5 in [27] for the details. 

Table [1] shows the notation used in the description of numerical experiments in the following subsec- 
tions. 
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Table 1: Notation 



iter. 

row A, col A 

nnzA 

SDPobj 

POPobj 

#solved 



minRatio 

aveRatio 

maxRatio 

sec 

min.t 

ave.t 

max.t 



the number of iterations in SeDuMi and SDPT3 

the size of coefficient matrix A in the SeDuMi input format 

the number of nonzero elements in coefficient matrix A in the SeDuMi input format 

the objective value obtained by SeDuMi for the resulting SDP relaxation problem 

the value of / at a solution x retrieved by SparsePOP 

the number of the POPs which are solved by SDP relaxation in 30 problems. If 

both e^i^- and Cfgag are smaller than l.Oe-7, we regard that the SDP relaxation 

attains the optimal value of the POP. 

minimum value of Ratio defined in ()17p in 30 problems 

average of Ratio defined in (IT71) in 30 problems 

maximum value of Ratio defined in (I17|) in 30 problems 

cpu time consumed by SeDuMi or SDPT3 in seconds 

minimum cpu time consumed by SeDuMi or SDPT3 in seconds among 30 resulting 

SDP relaxations 

average cpu time consumed by SeDuMi or SDPT3 in seconds among 30 resulting 

SDP relaxations 

maximum cpu time consumed by SeDuMi or SDPT3 in seconds among 30 resulting 

SDP relaxations 



5.1 Numerical results for POP whose quadratic module is non- Archimedean 

In this subsection, we give the following POP and apply Adaptive SOS relaxation: 



inf 

x,y£M 



fi{x,y) 
h{x,y) 
h{x,y) 



= a;-0.5 > 0, 
= y-0.5 > 0, 
= 0.5-a;y > 



(18) 



The optimal value is —1.5 and the solutions are (0.5,1) and (1,0.5). It was proved in [231 131] that 
the quadratic module associated with POP (TTSl) is non-Archimedean and that all the resulting SDP 
relaxation problems are weakly infeasible. However, the convergence of computed values of Lasserre's 
SDP relaxation for POP (fT8| was observed in [31) . 

In [31], it was shown that Lasserre's SDP relaxation ([2]) for (flSl) is weakly infeasible. Since Adaptive 
SOS relaxation for p^ has less monomials for representing CTj's than that of Lasserre's, the resulting 
SDP relaxation problems are necessarily infeasible. 

However, we expect from Thorem [2] that Adaptive SOS relaxation attains the optimal value —1.5. 
Table m provides numerical results for Adaptive SOS relaxation based on (|13p. In fact, we observe from 
Tabled] that p*{r) obtained by SeDuMi is equal to -1.5, r = 7,8,9, 10. By SDPT3, we observe similar 
results. 



5.2 The difference between Lasserre's and Adaptive SOS relaxations 



In this subsection, we show a POP where Adaptive SOS relaxation converges to the optimal value strictly 
slower than Lasserre's, practically. This POP is available at |6 , whose name is "st_e08.gms". 

a;> 0, 

y>o. 



inf 

x,y&t 



2x 



fi{x,y) 

y- h{x,y) 

fb{x,y) 



xy-l/16>0, f2{x,y) 
x>0, h{x,y) 

y>0, f6{x,y) 



x~ 

1 
1 



r-i/4>o. 



(19) 



The optimal value is (3^6 - ^2)78 « 0.741781958247055 and solution is {x*,y*) = ((V6 - ^2)78, (\/6 + 
V2)/8). 

Tables [3] and H show the numerical results of SDP relaxations for POP HH) by SeDuMi and SDPT3. 
We observe that Lasserre's SDP relaxation attains the optimal value of ([T9|) by relaxation order r = 3, 
while Adaptive SOS relaxation attains it only at the relaxation order by r = 6. 
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Table 2: The approximate optimal value, 
values computed by SeDuMi and SDPT3 



cpu time, the number of iterations and DIMACS errors of 



r 


Software 


iter. 


SDPobj 


[sec] 


crrl 


err2 


err3 


err4 


err5 


crr6 


1 


ScDuMi 
SDPT3 


46 
37 


-5.9100801e+07 
-1.8924840e+06 


0.31 
0.57 


9.3C-09 
1.8C-07 


O.Oc+00 
O.Oc+00 


O.Oe+00 
l.lc-07 


O.Oe+00 
O.Oe+00 


-5.8e-02 
-1.3C-01 


4.1e-01 
9.3e-02 


2 


ScDuMi 
SDPT3 


38 
72 


-6.8951407e+02 
-1.1676106e+04 


0.29 
1.28 


6.1C-10 
2.4C-13 


O.Oc+00 
O.Oc+00 


O.Oe+00 
2.2C-02 


O.Oe+00 
O.Oe+00 


-9.0C-02 
-l.lc-01 


3.1e-01 
8.7e-02 


3 


ScDuMi 
SDPT3 


32 

77 


-4.2408507e+01 
-2.0928888e+00 


0.22 
1.43 


7.7e-10 
7.5C-04 


O.Oc+00 
O.Oc+00 


O.Oe+00 
l.lc-03 


O.Oe+00 
O.Oe+00 


-4.3C-02 
-2.3C-02 


1.6e-01 
4.1e-02 


4 


SeDuMi 
SDPT3 


35 
76 


-1.2522887e+01 
-1.8195861e+00 


0.30 
1.74 


5.3e-10 
5.0e-05 


O.Oe+00 
O.Oe+00 


O.Oe+00 
1.4C-04 


O.Oe+00 
O.Oe+00 


-7.4C-03 
-3.8C-03 


1.4e-01 
2.1e-02 


5 


SeDuMi 
SDPT3 


32 

86 


-3.5032311e+00 
-1.6015287e+00 


0.39 
2.65 


1.2e-09 
9.9C-06 


O.Oe+00 
O.Oc+00 


O.Oe+00 
2.8C-05 


O.Oe+00 
O.Oe+00 


-5.9C-03 
-4.5C-03 


l.le-01 
7.4e-03 


6 


SeDuMi 
SDPT3 


33 

86 


-1.8717460e+00 
-1.5025613e+00 


0.48 
3.43 


2.7e-09 
3.0C-06 


O.Oe+00 
O.Oe+00 


O.Oe+00 
1.8C-06 


O.Oe+00 
O.Oe+00 


-3.0C-03 
3.8C-04 


3.8e-02 
2.1e-03 


7 


SeDuMi 
SDPT3 


17 
21 


-1.5000064e+00 
-1.5000022e+00 


0.47 
1.18 


5.9e-08 
6.5C-08 


O.Oc+00 
O.Oc+00 


O.Oe+00 
7.0C-09 


7.1e-09 
O.Oe+00 


-4.5C-07 
-5.6C-07 


1.2e-05 
4.9e-06 


8 


SeDuMi 
SDPT3 


16 
25 


-1.5000030e+00 
-1.5000001e+00 


0.58 
2.03 


8.0C-08 
3.8C-07 


O.Oc+00 
O.Oc+00 


O.Oe+00 
3.1e-10 


8.6e-09 
O.Oe+00 


-2.3C-07 
-9.5O-09 


7.3e-06 
2.3e-07 


9 


SeDuMi 
SDPT3 


15 
21 


-1.5000023e+00 
-1.4999912e+00 


0.75 
1.95 


1.3e-07 
1.3C-05 


O.Oc+00 
O.Oc+00 


O.Oe+00 
1.8C-10 


1.3e-08 
O.Oe+00 


-2.4C-07 
-1.7O-06 


7.6e-06 
1.7e-06 


10 


ScDuMi 
SDPT3 


15 
17 


-1.5000015e+00 
-1.5003641e+00 


0.99 
1.89 


1.2e-07 
1.3e-05 


O.Oc+00 
O.Oc+00 


O.Oe+00 
6.2C-08 


l.le-08 
O.Oe+00 


-1.6e-07 
2.2e-04 


6.1e-06 
6.7e-04 



Table 3: Numerical results on SDP relaxation problems in Subsection 5.2 by SeDuMi and SDPT3 





Lasserre 


Adaptive SOS 


r 


Software 


(SDPobj, POPobJl e^bj.^feas 1 M) 


(SDPobj, POPobJl e^bj'^feas 1 [«<=<=]) 


1 


SeDuMi 
SDPT3 


(O.OOOOOe+00, 0.00000e+00| O.Oc+00, -1.0c+00| 0.02) 
(-1.16657C-09, 5.89142c-10| 1.8c-09, -1.0c+00| 0.14) 


(O.OOOOOe+00, 0.00000e+00| O.Oc+00, -1.0e+00| 0.02 ) 
(-1.166576-09, 5.89142e-10| 1.8c-09, -1.0e+00| 0.06) 


2 


SeDuMi 
SDPT3 


(3.12500e-01, 3.12500c-01 -9.5c-10, -8.4c-01 0.09) 
(3.12500e-01, 3.12500c-01 2.0c-09 , -8.4c-01 0.22) 


(2.69356e-01, 2.69356e-01 -1.7e-10, -9.3e-01| 0.09) 
(2.69356e-01, 2.69356e-01 l.le-09, -9.3e-01| 0.21) 


3 


SeDuMi 
SDPT3 


(7.41782e-01, 7.41782c-01 -2.0c-ll, -l.lc-09 0.15) 
(7.41782e-01, 7.41782e-01 2.0c-08, 0.0c+00| 0.26) 


(3.063126-01, 3.06312e-01 -l.le-09, -8.3e-01| 0.13) 
(3.063126-01, 3.06312e-01 4.6e-09, -8.3e-01| 0.25) 


4 


SeDuMi 
SDPT3 


(7.41782e-01, 7.41782g-01 l.le-10, -1.5e-09| 0.15) 
(7.41782e-01, 7.41782c-01 2.8e-09, 0.0e+00| 0.34) 


(7.298556-01, 7.29855e-01 -1.2e-07, -4.9e-02| 0.24) 
(7.298556-01, 7.29855e-01 2.5e-08, -4.9e-02| 0.36) 


5 


SeDuMi 
SDPT3 


(7.41782e-01, 7.41782c-01 8.3c-ll, -4.5e-10| 0.19) 
(7.41782e-01, 7.41782c-01 -6.3c-10, 0.0c+00| 0.72) 


(7.361956-01, 7.36194e-01 -9.5e-07, -4.2e-02| 0.33) 
(7.361956-01, 7.36195e-01 5.3e-08, -4.2e-02| 0.50) 


6 


SeDuMi 
SDPT3 


(7.41782e-01, 7.41782c-01 2.3c-ll, -6.1c-ll| 0.27) 
(7.41782e-01, 7.41782c-01 3.4c-10, 0.0c+00| 1.02) 


(7.417826-01, 7.41782e-01 -l.Oe-09, -6.6e-09| 0.20) 
(7.417826-01, 7.41782e-01 -4.7e-ll, 0.0e-|-00| 0.98) 



Table 4: Iter, 
and SDPT3 



numerr, and DIMACS errors for SDP relaxation problems in Subsection 5.2 by SeDuMi 





Lasserre 


Adaptive SOS 


r 


Software 


(errl, err2, crr3, err4, crr5, crr6) 


(crrl, err2, crr3, err4, err5, err6) 


1 


SeDuMi 
SDPT3 


(2.8e-17, O.Oe+00, O.Oe-^00, O.Oe+00, 5.7c-20, 5.7e-20) 
(8.7e-14, O.Oe+00, 1.6e-10, O.Oe+00, 1.8e-09, 2.2c-09) 


(2.8e-17, -O.Oe+OO, O.Oe+OO, O.Oe+OO, 5.7e-20, 5.7e-20) 
(8.7e-14, O.Oe+OO, 1.6e-10, O.Oe+OO, 1.8e-09, 2.2c-09) 


2 


SeDuMi 
SDPT3 


(3.3e-ll, O.Oe-fOO, O.Oe+00, 3.7e-ll, -5.96-10, -5.8e-10) 
(1.96-15, O.Oe+00, 1.6e-12, 0.0e-|-00, 1.26-09, 1.2e-09) 


(7.06-11, O.Oe+OO, O.Oe+OO, 4.0e-ll, -l.le-10, -8.5e-ll) 
(2.36-15, O.Oe+OO, 1.6e-12, O.Oe+OO, 7.26-10, 7.3e-10) 


3 


SeDuMi 
SDPT3 


(1.5e-12, O.Oe-fOO, 0.0e-|-00, 5.8e-12, -8.1e-12, -7.9c-12) 
(2.8e-12, O.Oe-fOO, 1.6e-12, O.Oe+00, 8.0e-09, 8.1e-09) 


(9.46-11, O.Oe+OO, O.Oe+OO, 6.1e-12, -7.0e-10, -6.6c-10) 
(4.16-11, O.Oe+OO, 2.4e-12, O.Oe+OO, 2.8e-09, 2.9c-09) 


4 


SeDuMi 
SDPT3 


(2.1e-10, O.Oe+00, O.Oe-f-00, 1.6e-ll, 4.3e-ll, 6.3c-ll) 
(7.7e-10, O.Oe+00, 8.6e-13, O.Oe+00, l.le-09, 1.4c-09) 


(5.2e-10, O.Oe+OO, O.Oe+OO, 2.2e-ll, -5.0e-08, -5.0c-08) 
(4.6e-09, O.Oe+OO, 5.5e-12, O.Oe+OO, l.Oe-08, l.le-08) 


5 


SeDuMi 
SDPT3 


(1.9e-10, O.Oe+00, O.Oe+00, 8.2e-12, 3.4e-ll, 4.8c-ll) 
(2.0e-08, O.Oe+00, l.le-16, O.Oe+00, -2.5e-10, 9.7c-14) 


(6.8e-08, O.Oe+OO, O.Oe+OO, 1.5e-ll, -3.9e-07, -3.9c-07) 
(2.5e-07, O.Oe+OO, 5.1e-13, O.Oe+OO, 2.1e-08, 1.5e-08) 


6 


SeDuMi 
SDPT3 


(l.le-10, O.Oe+00, O.Oe-fOO, l.le-12, 9.46-12, 1.6c-ll) 
(3.0e-08, O.Oe+OO, l.Oe-16, O.Oe+OO, 1.46-10, 1.2e-13) 


(l.le-lO, O.Oe+OO, O.Oe+OO, 5.6e-12, -4.2e-10, -4.1c-10) 
(1.2e-08, O.Oe+OO, 5.3e-17, O.Oe+OO, -1.96-11, 4.6e-14) 
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5.3 Numerical results for detecting the copositivity 

The symmetric matrix A is said to be copositive if x'^Ax > for all x £ M" . 
problem for detecting whether a given matrix is copositive, as follows: 



We can formulate the 



inf ■! x'^Ax : fi{x) := x^ > (i = 1, . . . , n),fn+i{x) := 1 - y^ Xj = 0, 



(20) 



If the optimal value of this problem is nonnegative, then A is copositive. In this experiment, we solve 30 
Table 5: Information on SDP relaxations problems in Subsection 5.3 by SeDuMi and SDPT3 





Lasserre 


Adaptive SOS 




n 


Software 


(#solvcd min.t, ave.t, max.t) 


(#solved 1 min.t, ave.t, max.t) 


(minR, aveR, maxR) 


5 


SeDuMi 
SDPT3 


(30 1 0.14 0.18 0.50) 
(30 1 0.40 0.44 0.85) 


(30 1 0.12 0.16 0.20) 
(30 0.34 0.42 0.53) 


(1.0, 1.0, 1.0) 
(1.0, 1.0, 1.0) 


10 


SeDuMi 
SDPT3 


(29 1 0.36 0.42 0.50) 
(29 1 0.73 1.00 1.48) 


(29 1 0.23 0.31 0.42) 
(30 0.66 0.88 1.23) 


(1.0, 1.0, 1.0) 
(1.0, 1.0, 1.0) 


15 


SeDuMi 
SDPT3 


(30 1 1.59 1.99 2.52) 
(29 1 2.91 3.40 4.73) 


(30 1 0.75 0.99 1.31) 
(23 1.58 2.04 2.80) 


(1.0, 1.0, 1.0) 
(1.0, 1.0, 1.0) 


20 


SeDuMi 
SDPT3 


(30 1 10.22 14.06 19.98) 
(26 1 11.40 16.23 19.73) 


(30 1 4.47 6.02 7.72) 
(1 1 6.65 8.64 11.32) 


(1.0, 1.0, 1.0) 
(1.0, 1.0, 1.0) 


25 


SeDuMi 
SDPT3 


(29 1 215.94 263.88 336.96) 
(20 1 51.53 64.31 77.35) 


(29 1 49.69 66.63 84.07) 
(4 1 26.91 36.06 44.74) 


(1.0, 1.0, 1.0) 
(1.0, 1.0, 1.0) 


30 


SeDuMi 
SDPT3 


(27 1 1970.59 2322.30 2930.30) 
(0 1 136.59 401.23 1184.76) 


(28 1 1031.91 1198.05 1527.01) 
(0 1 92.96 165.22 295.23) 


(1.0, 1.0, 1.0) 
(0.4, 1.0, 1.6) 



problems generated randomly. In particular, the coefficients of all diagonal of A are set to be y/n/2 and 
the other coefficients are chosen from [-1, 1] uniformly. In addition, since the positive semidefiniteness 
implies the copositivity, we chose the matrices A which are not positive semidefinite. 

We apply Lasserre's and Adaptive SOS relaxations with relaxation order r — 2. Table [5] shows the 
numerical results by SeDuMi and SDPT3 for (|20|) . respectively. We observe the following. 



• SDPT3 fails to solve almost all problems ^, while SeDuMi solves them for n = 20,25,30. In 
particular, Adaptive SOS relaxations return the optimal values of the original problems although 
it is no stronger than Lasserre's theoretically. 

• SeDuMi solves Adaptive SOS relaxation problems faster than Lasserre's because the sizes of Adap- 
tive SOS relaxation problems are smaller than those of Lasserre's. 

• SDPT3 cannot solve any problems with n = 30 by Lasserre's and Adaptive SOS relaxation although 
it terminates faster than SeDuMi. In particular, for almost all SDP relaxation problems, SDPT3 
returns the message "stop: progress is bad" or "stop: progress is slow" and terminates. This means 
that it is difficult for SDPT3 to solve those SDP relaxation problems numerically. 



5.4 Numerical results for BoxQP 

In this subsection, we solve BoxQP: 

inf {x'^Qx + (Fx : < Xi < I (i = 1, . . . ,n)} , 



(21) 



where each element in Q e S" and c e M" is chosen from [-50, 50] uniformly. In particular, we vary the 
number n of the variables in (j2ip and the density of Q,c. In this subsection, we compare Adaptive SOS 
relaxation based on Theorem [5] with sparse SDP relaxation [27] instead of Lasserre's. Indeed, when the 
density of Q is small, the BoxQP has sparse structure, and thus sparse SDP relaxation is more effective 
than Lasserre's. 

We observe the following from Table [H 

• Sparse SDP relaxation obtains the optimal solution for some BoxQPs, while Adaptive SOS relax- 
ation cannot. 
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Table 6: 
SDPT3 



Information on SDP relaxation problems in Subsection 5.4 with density 0.2 by SeDuMi and 





Sparse 


Adaptive SOS 




n 


Software 


(#solvcd min.t, ave.t, max.t) 


(#solved 1 min.t, avc.t, max.t) 


(miriR, aveR, maxR) 


5 


SeDuMi 
SDPT3 


(23 0.15, 0.24, 0.48) 
(23 0.20, 0.37, 2.48) 


(23 0.14, 0.23, 0.52) 
(22 0.19, 0.26, 0.34) 


(0.00072, 12.34638, 342.39518) 
(0.00072, 0.97463, 1.24265) 


10 


SeDuMi 
SDPT3 


(13 0.33, 0.55, 0.70) 
(12 0.28, 0.51, 0.62) 


(12 0.28, 0.40, 0.53) 
(12 0.22, 0.29, 0.39) 


(0.97227, 0.99609, 1.00000) 
(0.97227, 0.99609, 1.00000) 


15 


SeDuMi 
SDPT3 


(14 0.57, 0.95, 1.68) 
(14 0.54, 0.93, 1.22) 


( 3 0.42, 0.63, 0.85) 
( 3 0.43, 0.57, 0.76) 


(0.96590, 0.99172, 1.00000) 
(0.96590, 0.99172, 1.00000) 


20 


SeDuMi 
SDPT3 


(11 1.40, 2.57, 5.32) 
(10 1.41, 2.31, 3.55) 


( 0.80, 0.97, 1.27) 
( 0.55, 0.69, 1.01) 


(0.94812, 0.98422, 0.99978) 
(0.94812, 0.98422, 0.99978) 


25 


SeDuMi 
SDPT3 


( 7 2.57, 5.15, 10.03) 
( 6 4.60, 7.24, 12.46) 


( 0.95, 1.09, 1.42) 
( 0.59, 0.85, 1.31) 


(0.94333, 0.97591, 0.99923) 
(0.94333, 0.97591, 0.99923) 


30 


SeDuMi 
SDPT3 


(12 3.43, 15.60, 26.86) 
(10 8.02, 22.87, 38.42) 


( 1.27, 1.51, 2.02) 
( 0.94, 1.33, 1.67) 


(0.93773, 0.97542, 0.99843) 
(0.93773, 0.97542, 0.99843) 


35 


SeDuMi 
SDPT3 


(12 26.57, 67.79, 143.06) 
( 9 44.14, 80.48, 135.30) 


( 1.77, 2.15, 3.33) 
( 1.06, 1.83, 2.63) 


(0.93271, 0.97236, 0.99648) 
(0.93271, 0.97236, 0.99648) 


40 


SeDuMi 
SDPT3 


Not solved 
Not solved 


( 2.47, 2.89, 3.57) 
( 2.13, 3.13, 3.87) 


(- - 


~) 


45 


SeDuMi 
SDPT3 


Not solved 
Not solved 


( 3.58, 4.17, 5.51) 
( 4.12, 5.09, 6.35) 


(-, - 
(-, - 


~) 


50 


SeDuMi 
SDPT3 


Not solved 
Not solved 


( 5.30, 7.02, 9.48) 
( 5.19, 6.83, 8.34) 


(-, - 
(-, - 


—) 


55 


SeDuMi 
SDPT3 


Not solved 
Not solved 


( 8.75, 10.43, 12.23) 
( 8.31, 10.77, 13.60) 


(-, - 
(-, - 


:] 


60 


SeDuMi 
SDPT3 


Not solved 
Not solved 


( 12.21, 15.16, 19.59) 
( 12.62, 16.57, 22.44) 


(-, - 
(-, - 


~) 



Table 7: 
SDPT3 



Information on SDP relaxation problems in Subsection 5.4 with density 0.4 by SeDuMi and 





Sparse 


Adaptive SOS 




n 


Software 


(#solved min.t, avc.t, max.t) 


(#solved 1 min.t, avc.t, max.t) 


(minR, aveR, maxR) 


5 


SeDuMi 
SDPT3 


(24 0.14, 0.19, 0.28) 
(23 0.23, 0.27, 0.35) 


(22 0.13, 0.17, 0.27) 
(22 0.17, 0.23, 0.34) 


(0.98678, 0.99849, 1.00000) 
(0.98678, 0.99849, 1.00000 ) 


10 


SeDuMi 
SDPT3 


(19 0.28, 0.49, 0.77) 
(18 0.32, 0.53, 0.86) 


( 9 0.25, 0.35, 0.46) 
( 7 0.26, 0.33, 0.52) 


(0.95400, 0.98958, 1.00000) 
(0.95400, 0.98958, 1.00000) 


15 


SeDuMi 
SDPT3 


(13 0.76, 1.21, 2.50) 
(13 0.84, 1.32, 2.26) 


( 3 0.46, 0.56, 0.65) 
( 3 0.37, 0.54, 0.81) 


(0.95219, 0.98580, 1.00000) 
(0.95219, 0.98580, 1.00000) 


20 


SeDuMi 
SDPT3 


(11 2.10, 3.51, 5.45) 
(11 3.22, 5.61, 8.30) 


( 0.70, 0.79, 0.97) 
( 0.50, 0.73, 1.01) 


(0.94457, 0.97953, 0.99933) 
(0.94457, 0.97953, 0.99933) 


25 


SeDuMi 
SDPT3 


(11 6.65, 13.88, 24.32) 
(10 11.48, 21.00, 30.98) 


( 1.02, 1.13, 1.28) 
( 0.69, 1.03, 1.47) 


(0.92917, 0.96999, 0.99596) 
(0.92917, 0.96999, 0.99596) 


30 


SeDuMi 
SDPT3 


(14 27.25, 60.67, 108.22) 
(12 43.33, 66.25, 95.80) 


( 1.31, 1.62, 2.26) 
( 1.29, 1.71, 2.22) 


(0.92761, 0.97283, 0.99608) 
(0.92761, 0.97283, 0.99608) 


35 


SeDuMi 
SDPT3 


( 8 76.07, 328.08, 589.43) 
( 6 116.23, 218.61, 322.82) 


( 2.11, 2.42, 2.95) 
( 2.21, 2.87, 5.03) 


(0.93669, 0.96707, 0.99717) 
(0.93669, 0.96707, 0.99717) 


40 


SeDuMi 
SDPT3 


Not solved 
Not solved 


( 3.11, 3.54, 4.69) 
( 3.29, 4.50, 5.39) 


(-, - 
(-, - 


—) 


45 


SeDuMi 
SDPT3 


Not solved 
Not solved 


( 4.99, 5.79, 7.10) 
( 5.43, 6.89, 8.85) 


(-, - 
(-, - 


:| 


50 


SeDuMi 
SDPT3 


Not solved 
Not solved 


( 7.09, 8.47, 11.58) 
( 9.09, 11.30, 15.02) 


(-, - 
(-, - 


~) 


55 


SeDuMi 
SDPT3 


Not solved 
Not solved 


( 11.84, 14.34, 17.72) 
( 14.09, 18.30, 22.13) 


(-, - 
(-, - 


~) 


60 


SeDuMi 
SDPT3 


Not solved 
Not solved 


( 19.33, 24.23, 29.13) 
( 19.45, 22.96, 26.65) 


(-, - 
(-, - 


~) 
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Table 8: 
SDPT3 



Information on SDP relaxation problems in Subsection 5.4 with density 0.6 by SeDuMi and 





Sparse 


Adaptive SOS 




n 


Software 


(#solved min.t, ave.t, max.t) 


(#solved 1 min.t, avc.t, max.t) 


(miriR, aveR, maxR) 


5 


SeDuMi 
SDPT3 


(27 0.13, 0.22, 0.54) 
(26 0.21, 0.26, 0.33) 


(25 0.12, 0.17, 0.38) 
(25 0.18, 0.21, 0.29) 


(0.93673, 0.99543, 1.00000) 
(0.93673, 0.99543, 1.00000) 


10 


SeDuMi 
SDPT3 


(19 0.36, 0.68, 1.26) 
(18 0.37, 0.48, 0.72) 


( 6 0.33, 0.48, 0.79) 
( 6 0.25, 0.31, 0.40) 


(0.94709, 0.98678, 1.00000) 
(0.94709, 0.98678, 1.00000) 


15 


SeDuMi 
SDPT3 


(14 0.71, 1.52, 3.70) 
(14 0.77, 1.33, 2.04) 


( 6 0.42, 0.61, 1.01) 
( 6 0.34, 0.41, 0.51) 


(0.95463, 0.98581, 1.00000) 
(0.95463, 0.98581, 1.00000) 


20 


SeDuMi 
SDPT3 


(13 1.92, 5.18, 7.99) 
(11 2.25, 5.54, 8.21) 


( 2 0.72, 0.91, 1.56) 
( 2 0.52, 0.61, 0.75) 


(0.92378, 0.97521, 1.00000) 
(0.92378, 0.97521, 1.00000) 


25 


SeDuMi 
SDPT3 


(15 9.56, 29.31, 57.08) 
(12 15.55, 26.06, 40.61) 


( 1.03, 1.24, 1.94) 
( 0.75, 0.93, 1.19) 


(0.92768, 0.96827, 0.99715) 
(0.92768, 0.96827, 0.99715) 


30 


SeDuMi 
SDPT3 


(11 50.72, 168.53, 368.04) 
( 9 42.25, 90.31, 140.94) 


( 1.56, 1.97, 2.99) 
( 1.27, 1.50, 2.10) 


(0.93048, 0.96888, 0.99470) 
(0.93048, 0.96888, 0.99470) 


35 


SeDuMi 
SDPT3 


(12 510.67, 964.20, 1489.56) 
(11 217.87, 303.90, 366.57) 


( 2.52, 3.11, 4.27) 
( 2.16, 2.55, 3.09) 


(0.90892, 0.95875, 0.99301) 
(0.90892, 0.95875, 0.99301) 


40 


SeDuMi 
SDPT3 


Not solved 
Not solved 


( 3.77, 4.34, 5.77) 
( 3.37, 4.24, 5.12) 


(-, - 
(-, - 


:] 


45 


SeDuMi 
SDPT3 


Not solved 
Not solved 


( 6.08, 6.91, 8.33) 
( 5.63, 7.07, 9.33) 


(- - 


~) 


50 


SeDuMi 
SDPT3 


Not solved 
Not solved 


( 8.97, 10.66, 12.82) 
( 8.87, 10.59, 11.84) 


(-, - 
(-, - 


~) 


55 


SeDuMi 
SDPT3 


Not solved 
Not solved 


( 13.95, 17.13, 20.71) 
( 10.26, 13.64, 20.92) 


(-, - 
(-, - 


—) 


60 


SeDuMi 
SDPT3 


Not solved 
Not solved 


( 21.94, 25.42, 30.36) 
( 15.48, 19.66, 27.26) 


(-, - 
(-, - 


:] 



Table 9: 
SDPT3 



Information on SDP relaxation problems in Subsection 5.4 with density 0.8 by SeDuMi and 





Sparse 


Adaptive SOS 




n 


Software 


(#solvcd 1 min.t, ave.t, max.t) 


(#solvcd min.t, avc.t, max.t) 


(minR, aveR, maxR) 


5 


SeDuMi 
SDPT3 


(25 0.15, 0.19, 0.34) 
(25 0.22, 0.27, 0.37) 


(22 0.13, 0.17, 0.24) 
(22 0.18, 0.22, 0.29) 


(0.94896, 0.99548, 1.00000) 
(0.94896, 0.99548, 1.00000) 


10 


SeDuMi 
SDPT3 


(20 0.36, 0.54, 0.80) 
(20 0.40, 0.62, 0.99) 


(11 0.26, 0.37, 0.52) 
(10 0.29, 0.39, 0.59) 


(0.96388, 0.99365, 1.00000) 
(0.96388, 0.99365, 1.00000) 


15 


SeDuMi 
SDPT3 


(14 0.93, 1.67, 2.93) 
(12 1.29, 1.85, 2.63) 


( 1 0.50, 0.59, 0.71) 
( 1 0.42, 0.51, 0.71) 


(0.94514, 0.98537, 1.00000) 
(0.94514, 0.98537, 1.00000) 


20 


SeDuMi 
SDPT3 


(14 2.51, 5.22, 8.98) 
(12 4.50, 6.70, 9.35) 


( 2 0.66, 0.85, 1.15) 
( 2 0.56, 0.76, 1.13) 


(0.95261, 0.98061, 1.00000) 
(0.95261, 0.98061, 1.00000) 


25 


SeDuMi 
SDPT3 


(10 10.64, 23.57, 56.02) 
(10 14.13, 26.81, 44.75) 


( 1.13, 1.25, 1.52) 
( 0.87, 1.11, 1.66) 


(0.95060, 0.97500, 0.99997) 
(0.95060, 0.97500, 0.99997) 


30 


SeDuMi 
SDPT3 


(11 42.70, 156.60, 507.20) 
( 9 53.52, 104.12, 173.49) 


( 1.68, 1.89, 2.18) 
( 1.43, 1.88, 2.49) 


(0.94199, 0.96738, 0.99484) 
(0.94199, 0.96738, 0.99484 ) 


35 


SeDuMi 
SDPT3 


(15 185.51, 1000.24, 2158.08) 
(12 157.31, 337.69, 508.43) 


( 2.66, 2.89, 3.15) 
( 2.52, 2.99, 3.60) 


(0.92313, 0.96254, 0.99485) 
(0.92313, 0.96258, 0.99485) 


40 


SeDuMi 
SDPT3 


Not solved 
Not solved 


( 4.45, 4.89, 6.34) 
( 4.11, 5.22, 6.66) 


[:;: 


:| 


45 


SeDuMi 
SDPT3 


Not solved 
Not solved 


( 6.52, 7.63, 8.86) 
( 7.00, 8.05, 9.51) 


(~, ~ 


~) 


50 


SeDuMi 
SDPT3 


Not solved 
Not solved 


( 10.45, 11.70, 13.89) 
( 10.57, 12.65, 15.41) 


[:;: 


~) 


55 


SeDuMi 
SDPT3 


Not solved 
Not solved 


( 15.96, 19.55, 24.40) 
( 11.84, 16.07, 21.26) 


!:;: 


—) 


60 


SeDuMi 
SDPT3 


Not solved 
Not solved 


( 26.31, 32.04, 36.89) 
( 17.69, 22.33, 27.93) 


[:;: 


:J 


70 


SeDuMi 
SDPT3 


Not solved 
Not solved 


( 69.62, 91.01, 123.14) 
( 26.30, 34.00, 45.75) 


(~, ~ 


~) 


80 


SeDuMi 
SDPT3 


Not solved 
Not solved 


( 182.40, 218.82, 268.42) 
( 46.87, 52.48, 59.51) 


(~, ~ 


~) 


90 


SeDuMi 
SDPT3 


Not solved 
Not solved 


( 406.85, 478.44, 619.49) 
( 77.36, 91.34, 107.29) 


(~, ~ 


~) 


100 


SeDuMi 
SDPT3 


Not solved 
Not solved 


( 844.15, 943.74, 1138.27) 
( 130.50, 148.36, 172.25) 


j:;: 


:| 
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• Adaptive SOS relaxation solves the resulting SDP problems approximately 10 ^ 30 times faster 
than Lasserre's. 

• The values obtained by Adaptive SOS relaxation are within 10% of Sparse SDP relaxation, except 
for n = 5. 

5.5 Numerical results for Bilinear matrix inequality eigenvalue problems 

In this subsection, we solve the binary matrix inequality eigenvalue problems. 

where we define for /c G N, x G M^ and y G M"^: 

n m n m 

Bk{x, y) = X! X! BijXiyj + ^ BioXi + ^ Bojyj + Bqq, 

i=l j=l i=l j=l 

where Bij{i = 0,...,n,j = 0, ...,to) are k x k symmetric matrices. In this numerical experiment, 
each element of Bij is chosen from [—1, 1] uniformly. (|22l) is the problem of minimizing the maximum 
eigenvalue oi Bk{x,y) keeping Bk{x,y) positive semidefinite. 

We apply Lasserre and Adaptive SOS relaxations with relaxation order r = 3. Tables (TU] shows the 
numerical resuhs for BMIEP ^ with A: = 5, 10 by SeDuMi and SDPT3, respectively. 

Tabic 10: Information on SDP relaxation problems in Subsection 5.5 by SeDuMi and SDPT3 





Lasserre 


Adaptive SOS 




(n, m, k) 


Software 


(#solvcd 1 min.t, ave.t, max.t) 


(#solved min.t, ave.t, max.t) 


(miriR, avcR, maxR) 


(1, 1, 5) 


SeDuMi 
SDPT3 


(21 1 0.11, 0.16, 0.25) 
(21 1 0.28, 0.36, 0.42) 


(16 1 0.10, 0.17, 0.29) 
(16 1 0.26, 0.32, 0.41) 


(1.00000, 1.00103, 1.02352) 
(1.00000, 1.00103, 1.02352) 


(1, 1, 10) 


SeDuMi 
SDPT3 


(20 1 0.12, 0.16, 0.21) 
(20 1 0.32, 0.37, 0.47) 


(18 1 0.11, 0.16, 0.30) 
(18 1 0.26, 0.33, 0.44) 


(1.00000, 1.00018, 1.00450) 
(1.00000, 1.00018, 1.00450) 


(3, 3, 5) 


SeDuMi 
SDPT3 


(3 1.95, 3.49, 4.74) 
(3 4.81, 7.12, 8.77) 


(1 0.43, 0.63, 0.81) 
(1 0.67, 0.97, 1.14) 


(0.878394, 1.01520, 1.20254) 
(0.878394, 1.01520, 1.20254 ) 


(3, 3, 10) 


SeDuMi 
SDPT3 


(0 2.46, 3.89, 4.77) 
(0 5.51, 7.63, 8.95) 


(0 0.54, 0.69, 0.93) 
(0 0.88, 1.04, 1.16) 


(1.00000, 1.00407, 1.01243) 
(1.00000, 1.00407, 1.01243 ) 


(5, 5, 5) 


SeDuMi 
SDPT3 


(0 219.93, 350.02, 545.81) 
(0 160.89, 247.24, 298.97) 


(0 8.25, 10.99, 14.08) 
(0 4.45, 5.50, 6.97) 


(0.649823, 1.04081, 1.26310) 
(0.649823, 1.04081, 1.26310) 


(5, 5, 10) 


SeDuMi 
SDPT3 


(0 285.21, 420.27, 509.31) 
(0 217.48, 276.67, 309.27) 


(0 7.96, 10.53, 15.04) 
(0 4.34, 5.37, 6.66) 


(1.00000, 1.01445, 1.02818) 
(1.00000, 1.01445, 1.02818) 



We observe the following: 

• SDPT3 solves SDP relaxation problems faster than SeDuMi for [n, m) = (5, 5). 

• Adaptive SOS relaxation can solve the resulting SDP problems faster than Lasserre's. In particular, 
SDPT3 works efficiently for Adaptive SOS relaxation for BMIEP 



6 Concluding Remarks 

In this paper, we present a perturbed SOS theorem (Theorem [Ij and its extensions, and propose a new 
sparse relaxation called Adaptive SOS relaxation. During the course of the paper, we have shed some 
light on why Lasserre's SDP relaxation calculates the optimal value of POP even if its SDP relaxation 
has a different optimal value. The numerical experiments clearly show that Adaptive SOS relaxation is 
promising, justifying the need fro future research in this direction. 

Of course, if the original POP is dense, i.e., Fj contains many elements for almost all j, then the 
proposed relaxation has little effect in reducing the SDP relaxation. However, in real applications, such 
cases seem rare. 
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In the numerical experiments, we sometimes observe that the behaviors of SeDuMi and SDPT3 are 
very different each other. See, for example. Table [5] In the column of Adaptive SOS, SeDuMi solved 
significantly fewer problems than SDPT3. On the other hand, there are several cases where SeDuMi 
outperforms SDPT3. For such an example, see the sparse relaxation column of Table |8l This is why we 
present the results of both solvers in every table. In solving a real problem, one should be very careful 
in choosing the appropriate SDP solver for the problem at hand. 
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A A proof of Lemma [6] 

As we have already mentioned in Remark [71 Lemma |S] is slightly different from the original one in [71 
Lemma 4] . To show the correctness of Lemma |6l we use the following lemma: 

Lemma 14 (f^ Lemma 3]) Let B C R" be a compact set. Assume that nonem,pty sets Ci,...,Cp C 
{1, . . . , n} satisfy (RIP) and we can decompose f into f = fi + ■ ■ ■ + fp with fh S M[a;Cfc] {h — 1, . . . ,p). 
In addition, suppose that / > on B. Then there exists g^ G M.[xCh] ^'^^ gh > on B such that 

/ == 31 H +gp- 

We can prove Lemma [HI in a manner similar to 'T, Lemma 4]. We define Fr : M" — > R as follows: 

p 

h=l 

We recall that ipr.h — J^jec (^ ^ fj/^j)^^fj fo'" all h = 1, . . . ,p and r e N, and that Rj is the maximum 
value of \fj\ on B for all j = l,...,77i. It follows from the definitions of ^pr,h and Rj that we have 
V'r,/! ^ "fpr+i.h on B for all h = 1, . . . ,p and r e N, and thus we have Fr < Fr+i on B. In addition, we can 
prove that (i) on i? n iC, F^ — > / as r — > cx), and (ii) on B\ K, Fr ^ oo a.s r ^ oo. Since B is compact, 
it follows from (i), (ii) and the positiveness of / on B that there exists f G N such that for every r > r, 
Fr > on B. Applying Lemma [T4l to Fr, we obtain the desired result. 
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